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NONLINEAR CONSTRAINED OPTIMIZATION 


Neculai ANDREI! 


Abstract. For nonlinear constrained optimization a particle swarm optimization (PSO) 
method with a new penalty function method is presented. The penalty function includes a 
composite penalty factor, which introduce a linear progressive penalization subject to the 
values of the constraint violation into the current point. This new composite penalty function is 
used into the frame of the particle swarm optimization method. To improve the direct search 
of PSO a local coordinatewise search is used. The numerical experiments with this new 
penalization for 10 real nonlinear constrained optimization applications are reported. The 
obtained results are compared versus the well-known direct search methods COBYLA, DFL 
and the primal-dual interior-point algorithm with a filter line-search method IPOPT. The 
conclusion is that minimizing a special penalty function using particle swarm optimization 
method yield a competitive algorithm being more efficient versus COBYLA and DFL. 


Keywords: Particle Swarm Optimization; Penalty function; Composite function; COBYLA; DFL; 
IPOPT. 


1. Introduction 


Engineering nonlinear optimization problems are present in very numerous 
applications. These problems can be represented as: 
mun f (x) 

subject to: c(x) 50, 3=1,2.5m, (1) 

xEQ={x:l<x<u}, 
where f:R" >R,c,€R" >R,i=1,....m, are nonlinear functions, called 
functional constraints, and /,w<R" are lower and upper simple bounds on 
variables xeR", respectively. The above formulation is not restrictive because 
the inequality constraints of the form c,(x)=>0, can also be represented as 
—c;(x) <0, and an equality constraint, c,(x)=0, can be represented by two 
inequality constraints c,(x)<0 and —c,(x) <0. For solving these problems plenty 
of methods are known. These can be classified as gradient methods using the 
information given by the gradients and the Hessians of the functions f and 


c;,i=1,...,m, and direct methods using only the values of these functions along a 
sequence of points from Q. 
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The main methods based on derivative information (the first and the second order) 
can be classified as: penalty and augmented Lagrangian methods [16, 18, 32], 
sequential quadratic programming [5, 28], interior-point methods [28], filter 
methods [17], etc. One of the most common methods for nonlinear constrained 
optimizations is based on penalty function. In the penalty method the constraints 
are penalized by building an extended objective function, which is minimized by 
means of unconstrained optimization algorithms (see [4, 25, 28]). The penalty 
function aims to penalize infeasible solutions by increasing their fitness values 
proportionally to their level of constraints violation. Using the same principle, an 
extension of the penalty methods is given by the augmented Lagrangian methods 
[32, 18, 36]. 


Methods based on augmented Lagrangian often use gradient techniques for 
minimizing the augmented Lagrangian [3, 8, 9, 26], and rarely they are combined 
with heuristics that rely on a population of points where the Lagrangian function 
is evaluated [37, 41, 43]. 


On the other hand, the methods using only the values of the functions defining the 
problem are much diversified, the main ones being based on: genetic algorithms, 
particle swarm optimization, ant colony optimization, evolutionary algorithms, 
bacterial foraging algorithms, tabu search, electromagnetism-like mechanism, etc. 
An excellent survey on bio inspired optimization algorithms is given in [6]. 


In this paper we consider the Particle Swarm Optimization method for solving the 
nonlinear constrained optimization problems (1) using a special penalty method 
where a linear assignment of penalization is used. Plenty of particle swarm 
optimization (PSO) methods for solving nonlinear constraint optimization 
problems are known [21]. A particle-swarm optimization using modified BFGS 
updating [31, 33] for constrained nonlinear optimization was presented in [27]. 
Here, a hybrid algorithm which integrates the modified BFGS into the particle 
swarm optimization is used to minimize the augmented Lagrangian penalty 
function. The structure of this hybrid algorithm for the augmented Lagrangian 
minimization is taken from [4]. 


Another approach based on the augmented Lagrangian which solves a sequence of 
simple bound constrained sub-problems whose objective function penalizes 
equality and inequality constraints violation and depends on the Lagrange 
multipliers and a penalty parameter was given by Rocha and Fernandes [35]. 


Each sub-problem is solved by a population-based method that uses an 
electromagnetism-like mechanism introduced in [7]. An application of swarm 
optimization for nonlinear programming using the evaluation of infeasible 
particles is described in [12]. 


Another penalty function approach was described in [44]. 
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Here the constraints are penalized by means of a multi-stage assignment function. 
Other approaches for solving constrained nonlinear optimization problems have 
been suggested. For example, a hybrid multi-swarm particle swarm optimization 
method is proposed in [42], where the current swarm is partitioned into several 
sub-swarms and the particle swarm optimization is used as the search engine for 
each sub-swarm. In order to explore more promising regions of the search space 
differential evolution is incorporated in order to improve the personal best of each 
particle. A dynamic-objective particle swarm optimization for constrained 
optimization problems is presented in [24]. This method converts the original 
constrained optimization problem into a bi-objective optimization problem and 
then it enables each particle to dynamically adjust its objectives according to its 
current position in the search space. In [40] a measure of the infeasibility of a 
particle is computed using an exponential function which contains the logarithms 
of the modified constraints. For solving nonlinear constrained optimization 
problems, in this paper we consider a modification of the penalty function 
introduced in [44]. The penalty function used here is a composite function in 
which the constraints are penalized by means of a linear assignment function. In 
Section 2 we present the penalty function method used in this paper. Section 3 is 
dedicated to give the main ideas of particle swarm optimization method in 
conjunction to this new penalty function. In order to improve the best point 
generated by the PSO algorithm a procedure for local search around the best point 
is used. The numerical results of this method subject to a number of 10 nonlinear 
constrained optimization applications are presented in Section 4. Comparisons of 
our method versus the direct search methods: COBYLA [34] and DFL [23, as 
well as versus IPOPT [46, 47] illustrate the performances of our algorithm. 


2. The Penalty Function Method with a linear progressive penalization 


The penalty function method solves the nonlinear constrained optimization 
problem (1) by solving a sequence of unconstrained optimization sub-problems. 
The unconstrained optimization sub-problems are solved using the PSO method. 
In PSO a non-stationary penalty function with a multi-sage assignment function 
was introduced by [20] and used in [44] and [29]. In this paper, for problem (1), 
we modify this non-stationary penalty function and consider the following one: 


F(x) = f(x) +h(k) P(x), (2) 
where f(x) is the original objective function, h(k) is a variable coefficient 
dynamically modified at every iteration k, and P(x) is a composite penalty factor 
defined as: 


P(X) = AGon(qoye™, G3) 


i=l 
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where q,(x) = max{0,c,(x)}, i=1...,m. 


Observe that the function g,(x) gives a measure of the violation of constraint c, 
into the point x. 


If the constraint c, is satisfied, then it has no influence into the penalty factor, 1.e. 
P(x)=0 for every admissible x. O(q,(x)) is an assignment function which 
introduce a progressive penalization subject to the values of the constraint c,(x) 
into the current point. 


In order to emphasize the penalization of the violation of the constraints around 1, 
the function 7(q,(x)) is introduced as: 


l, if ¢g(x)<], 


2, if g,(x)21. cs 


V(q(x)) = 


The variable coefficient h(k) is dependent by the number of iteration k. Simple 
expressions of this coefficient can be h(k)=kvVk, or h(k) = Vk. 


Observe that the functions 6(.) and y(.) are dependent by the constraint functions 
defining the problem. 


The function 0(t) which is a penalization of the constraint c,(x) <0 is defined as: 


U, if t< 0.001, 
BG) sys Se , SPOON tet: (5) 
0.999 0.999 
U, if ¢>1, 


where u and U are the positive penalty coefficients (u<U ). 


In our numerical experiments we have considered u=10 and U =300. In other 
words, we consider a moderate linear evolution of the penalization coefficients 
associated to the constraints. 

Clearly, this continuous evolution of function 6(t) may be modified, by changing 
the values of parameters u and U, in order to accentuate or diminish the 
penalization of the constraints in P(x). 


The simple bound constraints defined by © are also introduced in function P(x) 
as functional constraints. Observe that P(x) is positive for any value of x. 


With these, the penalty function method for solving (1) consists of minimizing the 
penalty function (2), viewed as an unconstrained optimization problem, using the 
particle swarm optimization method, which we present in the next section. 
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3. The Particle Swarm Optimization Method 


Particle Swarm Optimization (PSO) is a stochastic global optimization method 
based on simulation of the social behavior [13, 21, 30]. This is a technique that 
can be likened to the behavior of a flock of birds or the sociological behavior of a 
group of people. PSO is a population based optimization technique, where the 
population is called swarm. Each particle represents a possible solution to the 
optimization problem. Along the iterations each particle accelerates in the 
direction of its own personal best solution computed so far, as well as in the 
direction of the global best position discovered by any of the particles in the 
swarm. The idea is that if a particle finds o better new solution, then all other 
particles in the swarm will move closer to it, thus intensively exploring the 
corresponding region. 


The PSO algorithm has three main steps: generating the particles’ positions and 
velocities, velocity update and finally position update. Suppose that the searching 


space is R" and the population of particles consists of npop particles 
Xi AX where X, ER", j=l,....npop. Each particle defines the position 


> “" npop? 


of the searching point from R", thatis X, =[X j,,...Xj,]- 


Therefore, at the k-th iteration, for all the particles the matrix of positions 
X* =[Xt,.... Xi plER””” is defined. 


The best previous position that is the position of the best value of the minimizing 
function corresponding to the particle j, at iteration k, is denoted as PF, 
j=1,...,npop. Clearly, Pe eR’. 


At the same time, the best position found by any other particle at iteration k, is 


denoted by G‘. G*‘ eR" and represents the best position found by all the 
population of particles. Every particle, at iteration k, is characterized by a 


velocity V; eR", j=1,...,npop. Therefore, for all population of npop particles, at 


iteration k, the matrix of velocities V‘ =[V‘,...,V,,.,1eR""”” can be defined. 
The PSO method is characterized by the following evolution of velocities: 

Vi" = | wi + cr! @ (PI -X}) +7 @(G* —X}) |, (6) 
for j=1,...,.npop, where: y is a constriction factor which is used to control and 


constrict velocities; w is the inertia weight; c, and c, are two positive constants, 


called the cognitive and social parameter respectively, also known as acceleration 
constants. 
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The constriction coefficient is determined as [10]: 


7 2 
a earl 


i eR™™ and i eR”? are two matrices with random numbers uniformly 


p=c,+c). 


distributed within the range [0,1]. © is product on components. 


The first term in (6), wV; is defining the current motion, the second one 
cr; ®(P*-X‘) is defining the particle memory influence and the last one 
CF; ®(G* — X/) is the swarm influence. 

The inertia factor w is for controlling the impact of the previous history of 
velocities on the current velocity. 


It is used to balance the global (wide-ranging) and local (nearby) search abilities 
and was introduced to improve the convergence rate of the PSO algorithm. 


A large inertia weight is more appropriate for global search facilitating the 
exploration (searching new areas), while a small one is more appropriate for local 
search facilitating the exploitation (fine tuning the current search area). 


A linear decreasing of the inertia weight over the course of search was proposed 
by Shi and Eberhart [38]. 


The parameters r/ and 7/ are used to maintain the diversity of the population of 
the swarm and they are uniformly distributed in the range [0,1]. 


The constant parameters c, and c, are not critical in PSO, but a fine tuning of 
them may result in faster convergence and alleviation of local minima. 

Better results are obtained by choosing a larger cognitive parameter c,, than a 
social parameter c,, but with c,+c, <4 [29]. 


The position update of each particle is computed using the velocity vector as: 


xia xk eye j=1,....npop. (7) 


The velocity update, the position update and the evaluation of the minimizing 
function in all particles are repeated until a desired convergence criterion is met. 


With these, the particle swarm optimization procedure for composite function 
F(x) can be summarized as follows: 
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Algorithm CPSO - Constrained Particle Swarm Optimization 


Step 1. | Initialize: c,, c,, W, Y%, Kya, and set k=1. 


max 


Generate a swarm of npop particles with random positions X, € R", 


Step 2. ; ct 
j=1,...,npop, and with random velocities V,; eR", j =1,...,npop. 


Step 3. | Evaluate the penalty function F(X ;), 7 =1,...,pop. 


Find the best particle X ,, 
Step 4. : s ; 
the index of the best evaluated particle. 


Modify the velocities and the positions of the particles by (6) and (7), 


where g =argmin{F(X ;): j=1,...,npop}, is 


Step 5. . 
respectively. 
Gye Evaluate the penalty function F for all particles and update the best 
HERO: position for each particle and its index g. 
Step 7 If k <k,,.,, then set k=k+1 and go to step 3; otherwise stop and output 


the best particle. 


Convergence analysis and stability studies on PSO have been reported by Clerc 
and Kennedy [10], Trelea [39], Yasuda et al. [45], etc. 


Research on performance improvements subject to the parameters variations and 
topological structures have been considered by Eberhart and Shi [14, 15], Li and 
Engelbrecht [22]. 


The CPSO algorithm gives only local solutions to problem (1). Therefore, 
possible this solution can be improved by local search. 


Since the value of the penalty function (2) is zero in any feasible point, a 
procedure for improving the solution given by CPSO is not based on the penalty 
function (2). 


A simple local search procedure we adopt here is a coordinatewise one applied to 
the best point. 

Let x be the best solution generated by CPSO algorithm. For each component i, 
x 1S assigned to a temporary point y. Then in y a movement of length 6>0 is 
carried out in coordinate i. If the point y satisfies the simple bounds and is 
feasible, and if the value of the objective function f is reduced, then x is 
replaced by y. When the point y is infeasible, or the value of the objective 
function computed in y is not improved, then it is rejected and another coordinate 
is tried. 


The local search stops when all coordinates were tried. 


The local search algorithm for searching around the solution given by CPSO is 
very simple and possible in some cases it can improve the performances of CPSO. 
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4. Numerical results 
Test problem suite 


In this paper we have considered a number of 10 applications of nonlinear 
constrained optimization. Table 4.1 presents the name of the application and 
references where they are described. 


Table 4.1. Name of the applications and references. 
ALKI Optimization of an alkylation process, [19, Problem No. 114. Page 123] 
CAM Shape optimization of a cam, [1, Application 5.21, page 117] 
MSP3 3-stage membrane separation [19, Problem No. 116. Page 124] 


MSP5 5-stage membrane separation process, [11], [2, Application 6, page 983] 
Optimal Reactor Design. [19, Problem No. 104. Page 113.], [1, 2013, 


PREC Application 7.1, page 161] 

PPSE Static Power Scheduling [19, Problem No. 107. Page 116.], [1, 2013, 
Application 6.4, page 144] 

TRAFO Transformer design [19, Problem No. 93. Page 108], [1, 2013, Application 


6.1, page 137] 
LATHE | Multi-spindle automatic lathe [1, Application 5.13, page 90] 
BRAKE | Optimal design of a disc brake [1, Application 5.8, page 83] 


SPRING Minimizing the weight of a tension/compression spring [1, Application 
5.3, page 73] 


Numerical results with CPSO-Constrained Particle Swarm Optimization 


In our numerical experiments we have considered the following values for the 
parameters in the particle swarm optimization: y =0.73, w=0.7, c,=2, c, =2. 
The penalty function (5) is defined with u=10 and U =300. The size of swarm, 
i.e. the number of particles in the swarm, is different for each application. 
Empirical studies shown that the number of particles can influence the results of 
the optimization. For some problems we see an improvement of performances as 
the size of the swarm is increased, while for others better results are obtained by 
smaller swarms. The number of iterations is limited to k,,,, =50000. For each 
application we have considered 50 independent experiments. An experiment was 
considered to be successful only if a feasible solution was obtained. The best 
solution corresponds to the minimum value of the objective function found in 
these 50 experiments. Table 4.2 shows the performances of CPSO algorithm, 
where n is the number of variables, m is the number of constraints (including the 
simple bounds on variables), size is the number of the particles in the swarm, iter 
is the number of iteration to get the best solution, vfo is the best value of the 
objective function obtained by the algorithm and vfoa is the best value of the 
objective obtained by local search. 
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Table 4.2. Performance of the CPSO algorithm. 


n m size Iter vfo vfoa 
ALKI 10 34 150 1172 -1760.975932 -1763.528164 
CAM 10 43 100 5923 -43.379410 -43.523641 
MSP3 13 41 50 394 97.558862 97.558862 
MSP5 16 53 20 367 183.211378 183.024078 
PREC 8 22 50 45 4.040709 4.022867 
PPSE 9 30 50 >50000 5055.897360 5055.897360 
TRAFO 6 14 10 111 135.761724 135.761724 
LATHE 10 36 25 498 -4433.63308 -4433.63308 
BRAKE 4 14 9 6 0.147958 0.147323 
SPRING 3 10 5 70 0.0126904 0.0126904 


The column vfoa in Table 4.2 represents the value of the objective function f 
obtained by local search around the best solution given by PSOA using different 
values for 5. Observe that in some cases the local search is not effective. 


Comparisons 


In Table 4.3 a comparison of CPSO versus COBILA [34], DFL [23] and IPOPT 
[46, 47] subject to the value of the objective function of the above constrained 
nonlinear optimization application is presented. COBYLA is a direct search 
optimization method that models the objective and the constraint functions by linear 
interpolation. Each iteration forms such a linear approximation at the vertices of a 
simplex and a trust region bound restricts each change to the variables. DFL solves 
the constrained nonlinear optimization problem by a sequence of approximate 
minimizations of a merit function where penalization of constrained violation is 
progressively increased. On the other hand, IPOPT is a primal-dual interior-point 
algorithm with a filter line-search method for nonlinear programming. 

Table 4.3. Comparisons: COBILA, DFL, CPSO, IPOPT 


COBILA DFL CPSO IPOPT 
ALKI -1550.38851 -931.120 -1763.528164 -1768.8069 
CAM -43.859947 -45.54602 -43.523641 -43.859947 
MSP3 97.587578 50.0 97.558862 97.587509 
MSP5 175.619466 209.96775 183.024078 174.786994 
PREC 3.951163 4.1431963 4.022867 3.951163 
PPSE 5055.0118 8063.5088 5055.897360 5055.01180 
TRAFO 135.075962 136.27182 135.761724 135.075962 
LATHE | -3434.70855 | -4429.152978 -4433.63308 -4430.08793 
BRAKE 0.1274 O.131321 0.147323 0.1274 
SPRING 0.0126652 0.0126897 0.0126904 0.0126652 
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Comparing CPSO versus IPOPT, from Table 4.3 we see that the sum of the 
absolute difference of the objective values obtained by these two algorithms for 
solving the above 10 applications is 19.0891. Comparing DFL versus IPOPT the 
sum of the absolute difference is 3932.965. Finally, comparing COBYLA versus 
IPOPT we get the difference 1214.630. Observe that subject to the value of the 
objective function the CPSO algorithm is closest to the primal-dual interior-point 
algorithm with a filter line-search method IPOPT. However, COBYLA was able 
to get the same value for the objective function as IPOPT in 6 out 10 applications 
considered in this numerical study. 


5. Conclusions 


PSO is one of the methods for solving in an approximate way the optimization 
problems. The algorithm is able to search an optimal value of a constrained 
minimizing function by comparing only the values of a penalty function 
associated to the original problem. The PSO algorithm contains some random 
factors. Therefore, the algorithm can be regarded as a stochastic one. In this paper 
for general nonlinear constrained optimization we introduced a new penalty 
function based on an assignment function which considers a linear progressive 
penalization subject to the violation of the constraints into the current point. The 
PSO method is used to minimize this penalty function. To improve the results of 
the algorithm a local coordinatewise search is considered around the best point 
obtained by the CPSO algorithm. Numerical experiments proved that this local 
coordinatewise search is dependent by the problem. Some remarks are in order. 
CPSO is very simple to be implemented and it is able to generate good enough 
approximate solutions. It is dependent by the first random number with which the 
searching process starts. Therefore better results are obtained by trying to solve 
the problem using different initial randomly selected population and by comparing 
the results. In general swarms of small sizes give better results, but this is 
dependent by the problem. The CPSO method is a competitive alternative to the 
very sophisticated gradient methods used for solving nonlinear constrained 
optimization problems. 
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